When an image is utilized for creating a content, a region against a user's intention may be present in the image. Specific exemplary regions against a user's intention may include scratch, contamination, and unintentionally captured person. When the image including such a region is utilized for creating a content without being restored, a reduction in content quality or a violation of privacy can be caused. In order to solve the problem, the user may use the image from which a region against the user's intention is removed and in which the removed region is restored without a feeling of strangeness, for creating a content.
The restoration will be described in more detail with reference to FIG. 20. FIG. 20 is an explanatory diagram showing a method for restoring a region against a user's intention. In FIG. 20, an image 2000 indicates an image before image restoration and an image 2002 indicates an image after image restoration. For example, when a region 2001 in the image 2000 is against a user's intention, an image user removes the region 2001 from the image 2000 and then restores the removed part according to a surrounding image, like 2003. The thus-restored image is utilized thereby to solve the above problem.
In order to restore an image in this way, a method for utilizing a set of pixels (which will be called patch below) included in a region in an image is effective. In the following description, a region to be restored in an image (that is, a region against a user's intention) is denoted as defect region and an image including a defect region is denoted as defect image. A region other than the defect region in the defect image (that is, a region in the image not including a defect region) is denoted as non-defect region. In the method utilizing a patch, a pixel value of a certain part of a non-defect region in an image is attached to a portion to be restored in a defect region, thereby restoring the image.
The method for restoring an image having a defect region by use of a patch will be described in more detail with reference to FIG. 21 and FIG. 22. FIG. 21 is an explanatory diagram showing an image before restoration. FIG. 22 is an explanatory diagram showing an image after restoration. There will be specifically described below a process in which an image 102 including a region 101 to be restored is restored to an image 105 shown in FIG. 22. A portion 103, a portion 106 and a portion 107 are present as defect regions and a portion 104, a portion 108 and a portion 109 are present as non-defect regions in the image 102 in FIG. 21. At first, a pixel value of the portion 104 as a non-defect region is corresponded to the portion 103 as a defect region in the image 102. The portion 104 is used to restore the portion 103, thereby obtaining the image 105 in which part of the defect region 101 is restored. The process is repeated until a defect region disappears, thereby obtaining a restored image 110 as in FIG. 22.
In the following description, a small region including a portion to be restored (that is, a defect region) in an image will be denoted as defect patch like the portion 103, and a small region utilized for restoring a portion to be restored in a defect patch will be denoted as reference patch like the portion 104. The reference patch is a region which corresponds to a range in which a defect patch is to be restored, and which does not include a defect region. The defect patch may be the portion 106 or the portion 107 as a candidate in addition to the portion 103. In the following description, the candidates will be described as defect patch candidates. Similarly, the reference patch may be the portion 108 or the portion 109 as a candidate in addition to the portion 104. In the following description, the candidates will be denoted as reference patch candidate.
A method described in Non-Patent Literature 1 is known as a technique utilizing a patch, for example. The method described in Non-Patent Literature 1 is such that a defect patch and a restoration order are determined based on a luminance gradient of a non-defect region, and the like, from among defect patch candidates defined only on the border with the defect region. A reference patch is determined between the determined defect patch and the reference patch candidates based on a similarity obtained from a difference between the pixel values in the non-defect region. In the following, the processings are sequentially performed to restore an image.
The method described in Non-Patent Literature 1 will be described in more detail with reference to FIG. 23. FIG. 23 is an explanatory diagram for explaining the method described in Non-Patent Literature 1. When a defect image such as an image 300 in FIG. 23 is present, in the method described in Non-Patent Literature 1, at first a defect patch to be restored is selected based on information such as luminance gradient of a non-defect region from among the defect patch candidates defining only on the border with the defect region. Specifically, a defect patch 301 to be restored is selected from among the patch 301, a patch 302 and a patch 303 as the patch candidates in the defect region based on the information such as luminance gradient of a non-defect region.
Then, similarities between the defect patch 301 and the reference patch candidates (such as a patch 304, a patch 305 or a patch 306) are calculated to select the patch 304 having the highest similarity. The similarity is defined such that a reference patch having a closest pattern to a non-defect region of the defect patch 301 has a high value. Specifically, the similarity is defined by the square sum of a difference in pixel value between the pixels in a non-defect region in the defect patch 301 and the pixels in the reference patch corresponding thereto. Last, the pixel value of the pixels in the non-defect region in the defect patch 301 is replaced with the pixel value of the selected reference patch 304 thereby to obtain an image 307. The above processings are repeated until a defect region disappears, thereby obtaining a restored image.
FIG. 24 is an explanatory diagram for explaining pixels in a reference patch corresponding to pixels in a defect patch. The pixels in the reference patch corresponding to the pixels in the defect patch mean that when a defect patch 2101 and a reference patch 2102 each configured of 5×5 pixels are present, a pixel corresponding to a pixel 2103 is a pixel 2104 and a pixel corresponding to a pixel 2105 is a pixel 2106.
A method described in Non-Patent Literature 2 is known as an improved method of the method in Non-Patent Literature 1. The method described in Non-Patent Literature 2 is such that a similarity between the selected defect patch and the selected reference patch is also considered in addition to a luminance gradient when a defect patch is selected.